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FOREWORD 

Higher education reforms over the last decade, resulting in the establishment of the 
European Higher Education Area, with new social demands and expectations, have 
greatly impacted quality assurance in higher education. As a follow-up activity to two 
previous surveys on external quality procedures^ ENQA conducted a third survey 
concentrating on good practice in external quality assurance in order to explore the 
changes that are taking place, as well as the future areas of focus and activities of quality 
assurance agencies. 

The report presents a collection of current features of good practice in external quality 
assurance within the new priorities formulated by ENQA and following the Leuven/ 
Louvain-la Neuve and Budapest/Vienna Communiques, highlighting the connection 
between the practices and the expected benefits for higher education institutions and 
stakeholders. In addition, the present report identifies practices that are expected to be 
implemented by quality assurance agencies, as well as areas where progress needs to be 
made, thus proposing a vision of the future of quality assurance procedures. The purpose 
of the report is also to promote technical and strategic networks among ENQA members 
addressing future challenges. 

We trust that the results of this project will benefit ENQA membership in providing 
new possibilities for partnerships and networking and that new steps will be taken in the 
field of quality assurance and quality enhancement in the higher education sector. 


Achim Hopbach 

President 

European Association for Quality Assurance in Higher Education (ENQA) 


1 Quality Procedures in the European Higher Education Area and Beyond - Second ENQA Survey (ENQA Occasional Paper 
No. 14, 2008) 

Quality Procedures in European Higher Education - An ENQA Survey, (ENQA Occasional Paper No. 5, 2003) 
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EXECUTIVE SUMMARY 

Variety and dynamism are two distinctive features of quality assurance in the EHEA. The 
second ENQA survey on quality procedures conducted in 2008 revealed that although 
the accreditation and evaluation of programmes are the most common approaches, 
followed at a significant distance by evaluation and accreditation of institutions and by 
audits, no single model for external quality assurance in European higher education is in 
place. Ninety percent of the quality assurance agencies were not confined to only one 
external quality assurance process but used more than one approach on a regular basis. 
At the same time, 75 percent of the agencies responded that they recently changed their 
approach or that they were about to do so. 

In order to learn more about current and future developments at national level, ENQA 
conducted a third project on "Quality procedures: visions for the future" which was the 
follow-up of two previous projects on quality procedures conducted in 2003 and 2008. 
This third project took stock of the development of quality assurance in the EHEA and 
revealed that variety and dynamism will remain distinct features. In addition, the survey 
gave indications on the priorities in the future development of quality assurance agencies 
in the EHEA. 

The agencies that are members or affiliates of ENQA consider the main priority to be 
the relationship between external quality assurance procedures and the development of 
mechanisms to enhance higher education. The relevant innovative practices currently 
implemented by the agencies show a high level of methodological variability, together 
with a certain confluence of processes around four forward-looking strategies. The 
selection and intensity with which these strategies are practiced vary from one agency to 
another. 

■ The first strategy involves promoting the QA systems managed by the HEIs and 
external review of their effectiveness in bringing about enhancement. This strategy 
is associated with a greater attention paid to institutional review and a resolve to 
optimise external review processes carried out by QA agencies, with the reduction 
of red tape and the main focus on results. 

■ To encourage comparability in external review processes and to foster the 
introduction of system-wide enhancements. The second strategy is based on 
benchmarking and the establishment of reference frameworks, for example, in 
learning outcomes and other areas of higher education. In relation to this point, QA 
agencies have identified the need for more international components, especially in 
relation to the quality of study programmes. 

■ Thirdly, there is the identification of excellence in higher education and the 
dissemination of good practice. This is an emerging area that stems from different 
lines of thinking, the aims of which are to generally improve the provision of study 
programmes, encourage innovation in teaching and promote the attractiveness of 
certain programmes at the international level. 
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■ The fourth strategy involves intensifying the monitoring and follow-up of the 
quality of programmes and institutions through the availability of figures, data and 
indicators, in order to continuously measure improvement. There is an increasing 
amount of available quantitative data on higher education, although further 
progress and improvement at the instrumental and interpretative level is necessary. 

These strategies will take into account that quality assurance of teaching and learning 
should always put the learner at the HEI into the focus of their activities. 

Complementary to these strategies, the agencies consider that progress needs to be 
made regarding the international recognition of evaluation and accreditation practices 
being implemented at national level. 

Last, but not least, it should be pointed out that the majority of agencies extend the 
scope of their activities sufficiently beyond the customary idea of what quality assurance 
and/or accreditation is. They increasingly provide QA services of an advisory nature to 
universities, policy makers and stakeholders, and in terms of methodological output. The 
knowledge and understanding that QA agencies have acquired from their establishment 
place them in a privileged position as think tanks. The actions implemented by QA 
agencies facilitate and enhance the engagement of stakeholders in the debate on the 
concept of quality in the higher education sector now and in the future.. 
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CHAPTER 1: 

INTRODUCTION 


1.1 BACKGROUND AND PURPOSE 

In the Prague Communique of 19 May 2001, the Ministers of the Bologna Process 
signatory states invited the European Network for Quality Assurance in Higher education 
(ENQA), together with the European University Association (EUA), the National Unions 
of Students in Europe (ESIB) and the European Commission, to collaborate in establishing 
a common framework of reference and disseminating good practices in the field of 
quality assurance in higher education. 

As a response to this mandate, a first comprehensive ENQA survey on Quality 
Procedures in European Higher Education was carried out by the Danish Evaluation 
Institute (EVA) in 2002. The evaluation methods used in Europe were detailed in a 
publication entitled Quality Procedures in European Higher Education - An ENQA 
SurveyT The reported concluded that a major progress towards convergence had been 
made in the basic methods and procedures even if national agencies had not had the 
same priorities. 

In May 2007, the Board of ENQA decided that ENQA should conduct, through 
project funding from the European Commission, a thorough update exercise of the 2002 
findings by collecting and analysing information on external quality procedures of ENQA 
members, affiliates and associates. Indeed, European quality assurance had developed 
significantly since 2002, being increasingly influenced by the Bologna process and the 
adoption of the Standards and Guidelines for Quality Assurance in the European Higher 
Education Area^ (ESG) in 2005, which brought about extensive changes that had an 
impact on quality assurance procedures applied in Europe. 

The second ENQA project on "Quality Procedures in European Higher Education" 
was launched in order to document and analyse the methodological state-of-the-art 
in general terms in all countries where ENQA members, associates and affiliates are 
located, laying emphasis on the types of quality assurance procedures used. In addition, 
part 3 of the ESG for the external quality assurance of agencies, as well as agencies' 
external review plans and attitudes towards the European Quality Assurance Register 
for Higher Education (EQAR) were integrated into the survey. The results of the second 
ENQA survey are presented in a publication Quality Procedures in the EHEA and Beyond 
- Second ENQA Survey^. 

In March 2010, the ministers of education participating in the Bologna Process 
officially launched the European Higher Education Area (EHEA). Two years after 
the completion of the second Quality Procedures Project (QPP), the Board of ENQA 
wished to map the changes that are taking place in quality assurance (QA) due to the 
establishment of the EHEA, where new social demands and expectations have been laid 
for higher education. 


2 European Network for Quality Assurance in Higher Education, Quality Procedures in European Higher Education - An 
ENQA Survey, Occasional Paper No. 5, ISBN 951-98680-8-9, Helsinki, 2003 

3 ENQA: Standards and Guidelines for Quality Assurance in the European Higher Education Area, ISBN 952-5539-04-0, 
Helsinki, 2005 

4 European Association for Quality Assurance in Higher Education, Quality Procedures in the EHEA and beyond - Second 
ENQA Survey, Occasional Paper No. 14, ISBN 978-952-5539-31-8, Helsinki, 2008 


Thanks to the funding of the European Commission through its Lifelong Learning 
Programme, the third Quality Procedures Project "Visions for the future" started in 
November 2010. Its objective is to collect features of good practice within the new 
priorities formulated by ENQA and following the Leuven/Louvain-la Neuve and 
Budapest/Vienna ministerial Communiques, with a view to explore practices that are 
expected to be implemented by the QA agencies in the near future and to map exiting 
developed procedures. 

This project is a follow-up activity to the two previous surveys on quality procedures 
mentioned above. However, the focus of this third QP project has been on a development 
oriented approach, on the future of external quality procedures and new practices. The 
aim of the project is to provide results that can be exploited by stakeholders interested in 
enhancing a quality culture in the field of higher education. 

The present final report will be presented at the meeting of the European ministers 
responsible for higher education in Bucharest in April 2012. 

1.2 METHOD USED AND PROCESS 

The project was divided into four phases. The first task undertaken by the Project 
group was to design and develop the third Quality Procedures survey (see Annex). 

The questionnaire was administered online through the SurveyMonkey tool and made 
available on the web. The link to the questionnaire was then circulated to the ENQA 
member agencies, associates and affiliates^ 

Since the objective of the survey was to promote dialogue with different 
representatives on the relevance and impact of different QA procedures and their future 
evolution, the respondents were strongly encouraged to complete the questionnaire in 
consultation with as many members of their Agency (staff, Board/Council members, 
etc.) and stakeholders as possible. 

The second phase of the project consisted in the analysis of the responses. The survey 
was divided into three sections covering: 

i. Introduction relating to general information about the agency and its relationship 
with ENQA; Methodology used for completing the questionnaire (5 questions); 

ii. Current and future practices in external QA in relation to the ENQA priorities and 
other complementary elements (32 questions); 

iii. Individual features of good practice implemented or expected to be applied in the 
near future (7 questions). 

The project group was divided into three groups of two pairs who were assigned to 
analyse the procedures according to ENQA priorities and emerging trends. 

The third phase resulted in the production and release of this report. The findings 
presented give a map of future visions on QA procedures for thoroughly chosen priorities, 
highlighting the identified expected benefits for HE institutions and stakeholders. 

The purpose of the present report is not to give a comprehensive picture of future 
practices, but rather to analyse the data gathered. Such data is the result of a self- 
assessment and self-reflection of the agencies in the sense that the answers reflect 
the agencies' opinions about their own approaches, trends and innovative quality 
procedures. 


5 Since October 2011, the associate and affiliates statuses have been merged into one single category. All former associate 
and affiliate bodies are now Affiliates. 
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Finally, the results of the survey will contribute to the fourth phase of the project, 
which is the creation of a web resource for agencies' quality procedures. This database 
on the ENQA website will gather good and efficient quality procedures to be shared 
among QA professionals in Europe and beyond. 

1.3 ORGANISATION OF THE PROJECT 

The Catalan University Quality Assurance Agency (AQU Catalunya) took the lead and 
responsibility of the project on ENQA's behalf. A Project Group was set up to conduct 
the survey and draft the report. The ENQA Secretariat conducted the technical and 
administrative responsibility of the project. 

The project group was composed of experts representing five quality assurance 
agencies, including three ENQA Board members and two representatives of the Central 
and Eastern European Network of Quality Assurance Agencies in Higher Education 
(CEENQA). Full ENQA members were chosen to make sure that the agencies fulfil the 
ESG framework in their activities. The composition of the group also ensured a fair 
regional distribution within Europe. 

■ Josep Grifoll, AQU and ENQA Board member. Chair of the project 

■ Achim Hopbach, GAC and ENQA President 

■ Helka Kekalainen, FINHEEC and ENQA Vice-President 

■ Christina Rozsnyai, Secretary General of CEENQA 

■ Todor Shopov, Vice President of CEENQA 

■ Nathalie Lugano, ENQA Secretariat, Secretary 

The ENQA Secretariat was responsible for the production and publication of this report. 

1.4 RESULTS OF THE SURVEY 

A total number of 37 responses have been received. However, nine responses have not 
been considered due to a lack of substantial answers to the questions. Therefore, the 
findings presented in this report are based on 28 responses (from 19 countries) to the 
questionnaire, which was distributed to all ENQA members, affiliates and associates in 
March 2011 and had as a deadline 15 April 2011. 

In cases where responses were not received from all respondents, the total given may 
be less than 28. For some questions, respondents had the opportunity to tick as many 
boxes as they wished. In these cases the total given may be more than 28. 

Respondents had the option to request that the information they gave be reported 
anonymously in the final report. Six respondents chose this option. The name and the 
country of those organisations are therefore omitted in the report itself. 
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INTRODUCTION AND BACKGROUND INFORMATION ABOUT RESPONDENTS 

As shown in table 1^, a total of 28 organisations responded substantially to the survey. 


NAME OF AGENCY 

COUNTRY 

Armenian National Center for Professional Education Quality Assurance (ANQA) 

Armenia 

Austrian Agency for Quality Assurance (AQA) 

Austria 

Austrian Accreditation Council (CAR) 

Austria 

Austrian FH Council (FHR) 

Austria 

European Council for Business Education (ECBE) 

Belgium 

National Evaluation and Accreditation Agency (NEAA) 

Bulgaria 

Agency for Science and Higher Education (ASHE) 

Croatia 

The Danish Evaluation Institute (EVA) 

Denmark 

The Accreditation Institution 

Denmark 

Estonian Higher Education Quality Agency (EKKA) 

Estonia 

Finnish Higher Education Evaluation Council (FINHEEC) 

Finland 

Evaluation Agency Baden-Wuerttemberg (evalag) 

Germany 

Hellenic Quality Assurance Agency for Higher Education (HQAA) 

Greece 

Hungarian Accreditation Committee (HAC) 

Hungary 

Higher Education and Training Awards Council (HETAC) 

Ireland 

Irish Universities Quality Board (lUQB) 

Ireland 

Foundation “Higher Education Quality Evaluation Centre” (HEQEC) 

Latvia 

Accreditation Qrganisation of Flanders and the Netherlands (NVAQ) 

Netherlands/Flanders 

Agencia de Avaliagao e Acreditagao do Ensino Superior (ABES) 

Portugal 

National Accreditation Agency of the Russian Federation (NAA) 

Russia 

National Center of Public Accreditation (NCPA) 

Russia 

Agency for Higher Education Quality Assurance and Career Development (AKKQRK) 

Russia 

Commission for accreditation and quality assurance (CAQA) 

Serbia 

Agency for Quality Assurance in the Galician University System (ACSUG) 

Spain 

Catalan University Quality Assurance Agency (AQU Catalunya) 

Spain 

Agencia Canaria de Evaluacion de la Calidad y Acreditacion Universitaria (ACECAU) 

Spain 

Swiss Center of Accreditation and Quality Assurance in Higher Education (QAQ) 

Switzerland 

Quality Assurance Agency for Higher Education (QAA) 

UK 


Table 1: Agencies participating in the Survey 

A majority of Full members answered the questionnaire (17), followed by Associates (7), 
Candidate members (3) and Affiliates (1). 


6 The names of the agencies in table 1 and in the report are referred to as they where written by those agencies to the 
survey. Some agencies are named by their national names while other agencies' names are translated in English. 
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METHODOLOGY USED FOR FILLING THE OUESTIONNAIRE 

Involvement of QA agency staff 

Only two agencies did not consult their staff to fill in the questionnaire. It is therefore 
satisfactory to see that, in the wide majority of cases, the responses to the survey are 
the result of a cooperation between all staff members, or a restricted number of them. 
Most of the time, such participation was done through meetings or interviews organised 
specifically for the purpose of completing the survey. In some cases, a draft was 
compiled beforehand and served as a basis for discussion. In general, both management 
and technical staff have been involved. 

Involvement of the Agency Board/Council 

About one third of agencies (7 out of 24) have not consulted their Board/Council. 

The remaining ones have approached all or some of the Board members upstream of 
submitting the questionnaire. 

The Executive board of NVAO was involved at the end of the agency's consultation 
process to approve the staff answers to the questionnaire. One agency did not directly 
consult the Board, but used their opinions (previously expressed during Board meetings) 
as a basis for completing the questionnaire. 

Involvement of other agents (e.g. stakeholders) 

Half of the agencies have asked other agents to collaborate (such as the National Council 
for Higher Education, external experts, ministries, universities, students, the Academic 
Infrastructure Evaluation Sounding Board in the UK). They were consulted through 
interviews, email or focus group discussions. Quality Directors from across the Irish 
university sector, were given an opportunity to contribute to the submission through 
the Irish Universities Association Quality Committee (lUAQC). AQU interrogated many 
stakeholders, but only one university answered and was, in general, of the same opinion. 
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CHAPTER 2: 

CURRENT QA PROCEDURES IN 
RELATION TO ENOA PRIORITIES 
AND OTHER COMPLEMENTARY 
ELEMENTS 


2.1 STAKEHOLDERS AND EXPECTED BENEFITS OF OA 

The agencies were requested to identify a maximum of three relevant QA procedures 
currently implemented and that are connected with the priorities of ENQA. According 
to the information provided, most agencies cover all institutions in the national system. 
Private institutions are less covered than public institutions. 

PROCEDURE IMPLEMENTED IN COOPERATION WITH ANOTHER AGENCY 

In most cases, agencies implement the procedure by themselves, without cooperating 
with another QA agency. 

PARTICIPATION OF EXTERNAL STAKEHOLDERS IN THE IMPLEMENTATION OF 
THE PROCEDURE 

In the three mentioned procedures, students are the most involved stakeholders, 
followed by employers and public authorities. The trade unions are the worst represented 
stakeholders. They participate in the procedures of less than 10 agencies. 

This draws a clear picture about stakeholder involvement in QA procedures. 
Participation of students and employers seems to be a standard in Europe 
notwithstanding the fact that four agencies do not use employers as peers out of which 
one (Associate) does not involve stakeholders at all. 

EXPECTED BENEFITS FOR STAKEHOLDERS 

In line with the findings in the previous paragraph, the expected benefits for stakeholders 
differ significantly. 

Generally speaking, "Improvement of public information of HE" is considered to be 
the most important benefit for all stakeholders. An average of 17 respondents name this 
as a benefit for students, 15 as a benefit for employers, and 14 as a benefit for public 
authorities. Since Trade Unions are not considered as an important stakeholder, it does 
not come as a surprise that an average of 10 agencies consider this as a benefit for this 
stakeholder group. 

The second important benefit at a significant distance from the first is "Improvement 
of teaching and learning processes". Unsurprisingly, this item ranks highest with the 
students (average of 19 responses), employers follow second with 13 responses, public 
authorities with nine and trade unions with four responses. 

The benefit that is considered least important by the respondents is "Better 
connection between social interests and HE outcomes". Although this is considered to be 
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the second important benefit for employers and public authorities, the item ranks very 
low regarding the other stakeholders. 

Also, only six (average) respondents felt that this would be of benefit for trade unions 
which again gives evidence about the fact that trade unions are not considered as 
important stakeholders. 

If the analysis turns the perspective round and focuses on the various stakeholders, 
it is obvious that respondents consider students as the most important beneficiaries 
of institutional and programme oriented quality assurance. With 63 hits across the 
various benefits they rank before employers who cast 57 hits. At a large distance public 
authorities follow with 55 hits and trade unions with only 35 hits. 

In the case of students, "Improvement of public information of HE" as well as 
"Improvement of teaching and learning processes" are considered to be the most 
important benefits, whereas the others cast more or less the same number of responses 
which are significantly less than the two most important. 

The same pattern of two major important benefits appears with the employers. 
However, next to the public information function they seem to benefit more from the 
better connection between "social interests and HE outcomes". 

As regards public authorities, it is not surprising that "Improvement of public 
information of HE" is considered to be the most important benefit. The others rank 
substantially lower. 

In the case of trade unions, one must say that agencies seem not to consider them 
as important stakeholders at all; at least they cannot see that they really benefit from 
quality assurance procedures. If at all, it would again be the public information function 
of quality assurance. 

In conclusion, one can say that, in line with the former outcomes, it is not surprising 
that public information on HE is considered to be the most important benefit for the 
two most important beneficiaries, i.e. students and employers. What is surprising is 
that the developmental aspect of quality assurance, which translates into the benefit 
of "Development of new indicators about the quality of HE; Enhancement of HE 
organisations and structures", is considered as an important benefit only for public 
authorities for the third procedure mentioned. 

ACTION FOR PROMOTING BENEFITS FOR INTERNATIONAL STAKEHOLDERS 

The responses to this question make it quite clear that specific actions for promoting the 
benefits to international stakeholders are not well developed, other than collaboration in 
international networks or participation of international peer experts. 

EXPECTED BENEFITS FOR HEIS 

The responses give a clear picture which corresponds to the earlier answers: HEIs are 
considered to benefit most in terms of "Development of the primary responsibility of 
HEIs on QA" which was nominated by 13 respondents. Again, this clearly shows that, 
in the field of quality assurance, agencies see themselves as primarily supportive of 
HEIs. In line with this and considering the three procedures altogether, "Enhancement 
of HE organisations and structures" and "improvement of public information on HE" are 
considered to be the second most important benefit for HEIs. 

What must concern is the fact that agencies consider themselves not to contribute 
to the promotion of innovation in HE. At least this is seen as the second least important 
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benefit for HEIs, only topped by "better connection between social interests and HE 
outcomes". These results can also mean that agencies think that HEIs do not consider 
these items as important functions. 

2.2 POLICY EVALUATION 

ACADEMIC MOBILITY AND ACCESS, RESPECTIVE INSTRUCTIONS FROM THE 
MINISTRY/NATIONAL AUTHORITIES 

Although the topics of student mobility and access to higher education are on top of the 
political agendas, there is no clear indication that ministries would call upon agencies 
to evaluate the success in these fields or specifically focus on these topics in the 
procedures. 

The picture as regards instructions from ministries is quite clear: various agencies take 
into account the issues related to mobility of students (such as recognition of modules, 
etc.) and access to HE, but the ministries do not stipulate specific or even additional 
quality criteria for these fields. In most cases, governments implement strategies or 
framework conditions for the enhancement of academic mobility and equitable access 
to HE and such strategies are simply followed by agencies. Two thirds of the responding 
agencies state that there are no direct instructions to develop procedures for promoting 
academic mobility. In most cases, equitable access to HE is the prerogative of the 
government which plays an important role by providing a legislative framework in this 
respect. Equitable access is therefore often not part of the quality assurance agencies' 
function. 

2.3 PRACTICES OF QA AGENCIES ACCORDING TO ENOA PRIORITIES 

This section is based on three questions addressed to agencies concerning their external 
quality assurance. 

a. Which area does your agency cover (i.e. in which area does it conduct external QA 
evaluation)? 

b. Which areas do you consider as a priority in your Agency? Rate each area 1 to 5. 

c. Considering the priorities of ENQA, in which area are you currently most effective 
and successful? 

The three questions were responded by all agencies (28). 

The questions were linked to nine ENQA priorities on external quality assurance. 
Those priorities were inspired by the ENQA Position Paper on Quality Assurance in the 
EHEA^. 

The analysis of the responses compares what is considered as a priority for the QA 
agencies with the actions taken to advance them. The result of this analysis is a first 
picture of possible directions on external QA in Europe that is complemented with the 
identification of relevant and advanced practices described by QA agencies in the survey. 

The survey identifies three different groups of priorities. The first group encompasses 
"QA and enhancement mechanisms", "QA and learning outcomes" and "QA and public 
information". 


7 ENQA Position Paper on Quality Assurance in the EHEA in view of the Leuven and Louvain-la-Neuve meeting of 
ministers responsible for higher education of 28-29 April 2009. ENQA. http://www.enqa.eu/files/ENQA_position, 
paper%20%283%29.pdf 
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It is noteworthy that the majority of QA agencies agree that the areas of this first group 
are important or very important priorities. In fact no agency answered that these areas 
were not at all a priority. 

A second group of ENQA priorities comprises "Excellence in higher education" "QA and 
employability" and "Qualifications Frameworks and QA". The areas of this group are also 
mostly considered as important or very important priorities. Contrary to the first group, 
some QA agencies reported that these were not at all a priority for them. For example, 
two agencies mentioned that the area of excellence in higher education and the area of 
qualification frameworks were not at all a priority and one agency answered likewise for 
"QA and employability". 

Finally, a third group of priorities is composed of "QA in online education", "lifelong 
learning and QA" and "QA and transnational programmes". In this group, the areas 
are perceived in a completely different way from the first two groups. In fact, there is 
no unanimity in ranking those areas as priorities, although a significant number of QA 
agencies cover those areas in their external QA evaluation procedures, for example 
transnational education. Moreover, a high number of agencies ranked those areas 1 or 2 
(not a priority) in a scale of 5 grades. This is the case for "QA in online education" with 
48 percent of the agencies and for "QA and transnational programmes" with 40 percent 
of the agencies. 

Among these three groups, the three highest priorities are "QA and enhancement 
mechanisms" (96.4%), "QA and public information on HE" (53.6%) and "Excellence in 
HE" (44%). 

In line with the findings regarding the areas covered by the agency, it is not surprising 
that "QA and enhancement mechanisms" rank by far highest. 

Although almost all agencies also cover "QA and public information on HE" in their 
procedures, this Item ranks significantly lower as far as the priorities are concerned; only 
seven agencies nominated this item as a priority. This might be considered as a lack of 
awareness of the role of QA in the field of transparency. 

One could expect a high response rate also for the item "Excellence in HE", but there 
is a significant difference from the enhancement aspect, and surprisingly enough the 
item "QA and public information on HE" ranks even slightly higher that the notion of 
excellence. 

On the other side, those items which closely relate to teaching and learning and 
even to areas which are still new phenomena to some of the agencies, such as "QA of 
transnational programmes" and "QA in online education", rank very low in the priorities 
of agencies. 

The survey requested quality assurance agencies to define the concepts of excellence 
and lifelong learning. The interpretation of other concepts such as the employability or 
the priority areas was left open in the questionnaire, but they can be interpreted in the 
analysis of relevant and advanced practices. 
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2.3.1 FIRST GROUP OF PRIORITY AREAS 




Table 2: Comparison between priorities and 
activities performed by the agencies (in 
percentage) 


QA agencies having a most effective 
mechanism 


B Covered by QA agencies 
B Priority for QA agencies (4-5) 


Table 3. Intensity of the priorities 
(in percentage) 


Very important priority (5) 


■ 4 

■ 3 

■ 2 


B Not at all priority (1) 


a) QA and enhancement mechanisms 

This is clearly the top European external QA area in terms of priority, activities developed 
and perception of effectiveness by QA agencies. The great majority of agencies (26 out 
of 28) cover "QA and enhancement mechanisms". Taking into account the development 
of QA in Europe since the nineties, it cannot surprise that those agencies which (among 
others) conduct institutional oriented QA procedures have a developmental approach. 

The establishment of the European Model for QA in Higher Education, including 
follow-up processes and recommendations to HE institutions, was mainly thought to 
promote the implementation of improvement plans within institutions. To some extent, 
the use of the European Model for more than a decade can be seen as a solid explanation 
for the strong position of enhancement mechanisms and the perception the QA agencies 
have of it. 

When looking at the relevant practices on QA and enhancement mechanisms, it is 
interesting to see that QA processes are usually designed for different purposes; in other 
words, there are very few QA processes designed to specifically address enhancement 
mechanisms in higher education. This multipurpose orientation of QA processes 
together with the dynamics of the EHEA and the external QA strategies, in constant 
evolution, makes it difficult to measure the impact the QA processes of agencies have on 
institutional enhancement. 

The survey suggests some tendencies or methodology reformulations in the QA 
processes for enhancement objectives. For example, some agencies use the concept 
of effectiveness of internal QA at the institutions. The ESG and the development of 
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the institutional QA systems are both used as a background for that approach. This 
effectiveness is based on the expectation that the HE institutions have a key strategy in 
place (according to the ESG principles) demonstrating that their internal QA is well fitted 
for a systematic improvement of study programmes. 

The accreditation of effectiveness and/or the reduction of burden in external QA, 
in particular in programme accreditation processes, can be seen as an incentive for HE 
institutions. This approach underpins the autonomy of the HE institutions and introduces 
a new distribution and balance of QA tasks between the agency and the institution. 
NVAO states that "this approach leaves the teaching staff free to devote their attention 
to expert suggestions for improving relating to the core of their teaching rather than 
spending time on institutional aspects". 

Other agencies are working with the identification of reference points, e.g. subject 
benchmark statements. In fact, the definition of reference points goes with the 
description of new thresholds (goals) or the identification of good practices that are 
disseminated. 

In this regard, it is interesting to mention that half of the respondent agencies think 
there is a major need for further development of current international benchmarks 
for QA. Whereas the ESG seem to be a good tool for implementing processes for 
internal QA effectiveness and permanent enhancement, the identification of reference 
points probably needs a major progress of elements, such as national and European 
qualifications frameworks, currently in the process of development in some countries. 

A third approach in this area is aiming to connect QA and enhancement through the 
establishment of consolidated cyclical procedures. Some agencies established annual 
monitoring systems for study programmes. This strategy makes it easier to follow, in real 
time, the evolution of the programmes and promotes a constant use and development of 
quantitative indicators. This follow-up scheme is also used to record the actions taken to 
improve the study programmes. 

This scenario leads the reader to reflect on how the strategies of agencies can be 
organised to foster the quality enhancement of higher education. Which is the most 
effective balance between internal and external QA or between programme and 
institutional approaches? To what extend does the diversity of HE systems justifies a 
diversity of external quality assurance? 

b) QA and learning outcomes 

This second area is covered by three-quarters of the respondent agencies and is ranked 
as a very important priority. However, only half of the agencies report they are currently 
most effective and successful in this area. 

The prominence of the item "Learning outcomes" gives an indication of the impact of 
the Bologna process on institutional quality assurance. Whereas it would be natural for 
this item to rank high in programme oriented approaches, it is noteworthy that also in 
institutional and rather developmental approaches this feature of the Bologna process 
plays an important role. This shows how important this topic has become. However, 
the item "Qualifications Frameworks (QF) and QA" ranks last (together with QA of TNE 
programmes and Lifelong Learning), although they are closely linked. The reason might 
rather be the early stage of development and implementation of QF in national HE. 

The survey collects numerous examples of accreditation procedures. QA agencies are 
conducting ex-ante or ex-post programme assessments. One can assume that learning 
outcomes are included in those assessment procedures in one way or another. 
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From the description of the relevant and innovative practices identified in the survey, 
different approaches can be observed. 

First of all, an increasing Importance is placed on learning outcomes in the 
methodology. There are examples of evaluation tools in which the definition and 
validation of intended learning outcomes are a central focus. In fact, the validation 
process is linked with a further elaboration and/or permanent update of the National 
Framework of Qualifications. 

It is interesting to mention the description of subject benchmarks as a clear orientation 
to be used by HEIs in designing new study programmes, but also as a fundamental 
information for enriching National Framework of Qualifications. 

Another interesting proposal, perhaps not sufficiently explored, concerns the role of 
international experts In the process of learning outcomes validation. 

Further steps are starting to be developed. An agency is working on a procedure 
ensuring that learners may achieve the learning outcomes specified and that these are 
consistent with relevant standards including the National Framework of Qualifications. 

Finally, measuring individual achievement of learning outcomes at international level 
is experienced by NCPA which promotes an internet based-Olympiad in which students 
from different countries participate. It is a wide-scale assessment of achieved learning 
outcomes. 

In conclusion, the work of QA agencies seems to be concentrated in the validation 
of learning outcomes. The new frontier is now how to measure their achievement by a 
group of graduates from the same study programme or by the students individually. 

c) QA and public information on HE 

It is noteworthy that the "QA and public information on HE" also ranks in the first group. 
Although the majority of respective procedures are rather developmental oriented 
evaluation or audit procedures, they serve this rather accountability driven purpose of 
QA. This is a first hint to the transparency function of QA which has gained importance 
trough the last years and also a hint to the combination of different purposes to be served 
by a single QA procedure. 

An interesting question relates to the kind of information that is provided as a result 
of these procedures. Is it purely developmental oriented or rather about achieved 
standards? The question cannot be answered within this survey but at least it reveals 
that agencies consider the information important for serving the transparency function of 
QA. 

As for the area of learning outcomes, this third priority shows another gap between 
the number of agencies covering this field (71.4 percent) and the number of agencies 
feeling they have effective and successful mechanisms (42.9 percent). It is interesting 
to mention that, in some cases, this gap may have originated from legal frameworks 
restricting the publication of assessment reports. In fact, some countries are adapting 
their legal frameworks, as a consequence of the process of reviewing QA agencies 
against the ESG, in order to facilitate the publication of the results of the external QA 
activity performed by the agencies. 

According to the survey, it is possible to identify some directions for the future. 
Obviously, the accumulation of review reports makes it necessary to reconsider the 
access to the results. QA agencies are developing databases or catalogues of reviews 
and make them available on their websites. This contributes to the objective of increasing 
transparency and making public information easily accessible. 
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A second strategy regards the contents of the reports. Accessible summary of the 
reports or the introduction of scales to rank the outcomes of the reviews is implemented 
by some agencies. 

The inception of distinctive quality features in the external QA procedures is used 
to provide stakeholders with additional public information on HE. For example, the 
introduction of the quality of internationalisation for study programmes is done with 
the objective to facilitate an easier transfer of information between agencies and 
stakeholders, at national or international level. 

A complementary strategy concerns the publication of follow-up reports which aims 
at confirming actions undertaken as a result of the review. 

Besides the system-wide analysis, some agencies provide information about the 
national HE system; in particular, they link the evaluation outcomes with the National 
Framework of Qualifications. Greater transparency of the nature of the qualifications 
provided enables learners to make better and more informed choices. 

Finally, it is interesting to mention an emerging triangle consisting of external QA 
procedures of agencies, the use of quantitative indicators and easier public access to that 
information. An increasing use of quantitative and computerised information systems 
lead to the scenario where different stakeholders have access to the raw data and 
compose their own rankings. Moreover, an easier access to the data is generating new 
opportunities to deal with the quality of HE not only for QA agencies or ranking makers, 
but also for other civic society institutions. 


2.3.2 SECOND GROUP OF PRIORITY AREAS 



Excellence in QA and Qualifications 

higher education employability framework and 

QA 


Table 4. Comparison between priorities 
and activities performed by agencies (in 
percentage) 


QA agencies having a most effective 
mechanism 


B Covered by QA agencies 
B Priority for QA agencies (4-5) 


100 



QA 
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a) Excellence in higher education 

Almost half of the agencies answered that their external QA processes cover the area 
of excellence in higher education; 70 percent consider it as a very important or quite 
important priority. However, the answers reveal that excellence can be understood in 
various ways. 

Excellence of higher education is, for some, seen as the overall goal of all external 
QA mechanisms. All activities are oriented towards excellence in higher education. 
Excellence seems to be a general notion instead of being a distinctive feature of activity. 
Sometimes the word "excellence" is not used, but the agency's work on enhancement 
and the dissemination of features of good practice from review processes can be linked 
to striving towards excellence. After reaching the minimum standards, the idea of 
excellence is a way to motivate HEIs for self-improvement and further development. 

The counterpoint to this view of seeing excellence as an overall goal of external QA 
is the idea that external QA is a means to ensure a threshold level of quality is reached. 
About 25 percent of the agencies consider excellence not being a priority at all or a low 
priority. 

An agency defines excellence as the "possession of good quality to an unusual 
degree". For NCPA, excellence in HE is a level of quality which surpasses the threshold 
standards. Some agencies that gave information on the way they deal with the notion of 
excellence In their procedures mainly refer to the fact that they give recommendations. 

More specific examples identify ways to make the difference between average 
and excellent. In these approaches, excellence is a distinctive and particular feature. 

In institutional processes, outstanding practices or particularly good achievements 
are highlighted in the review reports in various ways. For EKKA and QAA, excellence 
is linked to the agency's work on enhancement, where features of good practice are 
recognised and disseminated. Wordings like "worthy of recognition", "best practice" 
and "commendation" are used to bring out excellence. The application of a four-point 
scale (excellence, good, sufficient, insufficient) on the level of programmes and on the 
level of standards of the framework is a systemic way to single out excellence. NVAO 
carries out assessments of Internationalisation at programme level. In the Netherlands 
and Flanders, successful higher education programmes can get recognition for the 
quality of their internationalisation by obtaining the hallmark "distinctive quality feature 
internationalisation". Excellence in students' achievements and learning outcomes is 
measured by NCPA in open international Internet-Olympiad in different subject fields. 

Besides indentifying excellence, agencies can steer HEIs towards excellence by 
setting the standards, definitions or frameworks for excellence. lUQB plays a central role 
in defining the national excellence criteria through the production of a series of good 
practice guidelines and also the development of a national framework for internal and 
external quality processes. Three agencies base their reviews on defined criteria for 
excellence (among them lUQB and HAC). 

The survey did not include questions related to whether financial incentives or rewards 
are connected to excellence. Despite that, ASHE explains that excellence and state 
funding will be connected when the new legislation is fully implemented: more funding 
will be allocated to excellent institutions. Another example of connection between 
excellence and funding is the selection of the centres of excellence in education. Based 
on international anonymous review of applications and a site-visit to selected units, the 
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agency then appoints 8-10 units to carry the title "centre of excellence in education" for 
three years. The ministry allocates performance-based funding to the rewarded units. 

Another agency is developing its assessment activity in a governmental programme 
aiming to promote honours programmes and also to stimulate experiments with a wide 
variety of teaching and learning methods. In this case, the identification of excellence 
allows excellent programmes to select students and charge higher tuition fees. It is 
expected that only a very limited number of programmes will opt/qualify for this right. 

b) QA and employability 

Fifty percent of the respondent agencies report that they are currently conducting 
external QA evaluations which cover the issue of employability. A significant number of 
agencies consider the topic as a priority but, again, there is room for improvement since 
only 25 percent of the agencies declare having most effective and successful activities on 
QA and employability. 

Among the relevant practices identified in the survey, NEAA is working in cooperation 
with HE institutions to develop an Employability Framework at HEIs which is expected to 
define criteria for institutional and programme evaluations. The framework highlights the 
skills and attributes that are valued by employers. 

The external QA system starts the inclusion of the local employability expectations 
as a criterion to be considered for ex-ante evaluations and follow-up procedures. In this 
regard, AQU Catalunya conducted a survey for graduates as a tool to generate data on 
the employability of study programmes and the adequacy of learning outcomes in the 
labour market. 

Finally, some agencies inform that representatives from the employers and trade 
unions are represented in their Board or are involved in controlling the quality of 
formation. 

c) Qualifications frameworks and QA 

Fourty three percent of the respondent agencies conduct processes connecting QA and 
Qualifications Frameworks, but only seven agencies reported they are currently most 
effective and successful in this area. Among those agencies, some work in the definition 
of benchmark statements, and others on the validation of new study programmes. 

This priority is well connected with the area of QA and learning outcomes. Considering 
that the approval of the Qualification Frameworks is not normally a task of QA agencies, 
the role the agencies play in this area seems to be the development of the use of national 
and European qualifications framework as tools for external QA procedures. This is the 
case for QA agencies validating new programmes, in particular for professional oriented 
study programmes. 

The use of international Qualifications Frameworks is, at least in the survey, appearing 
as a pending issue for the majority of agencies. They are used to work with national 
Qualifications Frameworks only. Is this scenario an opportunity for the labels or similar 
organisations comparing qualifications and learning outcomes at international level to fill 
the vacuum? 
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2.3.3 THIRD GROUP OF PRIORITY AREAS 


QA of Lifelong learning QA and online 

transnational and QA education 

programmes 


100 



QA of Lifelong learning QA and online 

transnational and QA education 

programmes 


Table 6. Comparison between priorities and 
activities performed by the agencies (in 
percentage) 


QA agencies having a most effective 
mechanism 

Covered by QA agencies 
Priority for QA agencies (4-5) 


Table 7. Intensity of the priorities 
(in percentage) 


B Very important priority (5) 

■ 4 

■ 3 

■ 2 


Not at all priority (1) 


a) QA of transnational programmes 

This priority is covered by 43 percent of the agencies participating in the survey, but it is 
not seen as a top ranked priority. Although there is a strong European political support 
for the establishment of this type of programmes, the demand of national HE systems for 
external QA (quantitatively and qualitatively speaking) is taking the attention away from 
QA for transnational programmes. 

In fact, only 19 percent of the respondents report this is a very important priority, and 
about the same number reports that the agency is effective and successful carrying out 
procedures in this area. 

The relevant practices in this area are a fair reflection of a certain vacuum at the 
level of individual QA agencies. In fact, the field is currently explored by networks of QA 
agencies (like the European Consortium of Accreditation) or other bodies. Nonetheless, 
two agencies reported relevant practices on validation or assessment of collaborative 
and/or transnational programmes. 

In the list of advanced practices, it is interesting to highlight the case of an agency 
developing national policy and criteria for accreditation/validation, delegation of 
authority, and making awards in the context of collaborative programmes, trans-national 
programmes and joint award programmes. 

At European level, national perspectives on exporting and importing higher education 
and quality assurance procedures are still to be developed. 
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Not strictly linked to this area of transnational programmes, but in the field of 
developing internationalisation of higher education, is the advanced practice in which QA 
procedures are addressed to identify the internationalisation of study programmes. 

Finally, it is interesting to mention that more than half of the agencies (56,5%) 
taking part in the survey think there is a major need for improvement in developing the 
international recognition of national QA processes. This is important to bear in mind 
when developing the European dimension of QA further. 

b) Lifelong learning and QA 

The development of specific QA strategies and procedures does not seem to match 
the increasing importance of the implementation of lifelong learning schemes in our 
societies. 

Although 40 percent of the respondents reported to conduct external QA evaluation 
of lifelong learning, this area is not ranked as a very important priority. Moreover, there 
is a minority of agencies feeling effective and successful in this area. Some of them are 
accrediting or certifying this type of education and, to some extent, demonstrating that 
continuing education is comparable to the "regular" study programmes. In one case the 
external QA procedure reported in the survey was to be considered as a good practice. 
That concept of lifelong learning indicates that learning should encompass the whole 
spectrum of formal, non-formal and informal learning. In order to enable the individual 
to learn throughout life, equal value should be given to all these forms of learning 
regardless of source, how it is achieved or when in life it is achieved. A major objective 
of the National Framework of Qualifications is to recognise all learning achievements by 
supporting the development of alternative pathways to qualifications (or awards) and by 
promoting the recognition of prior learning. 

Although there is a gap between the policies and QA implementation for lifelong 
learning, it is important to observe the definitions and interpretations of what the 
agencies mean by lifelong learning in their context. This is providing prospective themes 
in lifelong learning and new working fields for QA agencies. 

The 13 respondents who answered the question "what you mean by lifelong learning 
in your context" mainly understand ILL as degree and non-degree courses offered by 
HEIs. In the UK, ILL might involve work-based learning and continuing professional 
development. An agency gives a more detailed definition which mentions informal 
learning, different access paths, etc.: "the concept of ILL should encompass the whole 
spectrum of formal, non-formal and informal learning. Learning occurs in many contexts 
that include work, involvement in social and community activities, or learning through 
life experience generally. In order to enable the individual to learn throughout life, 
equal value should be given to all these forms of learning regardless of source, how It Is 
achieved or when in life it is achieved. A major objective of the National Framework of 
Qualifications is to recognise all learning achievements. It aims to do this by supporting 
the development of alternative pathways to qualifications (or awards) and by promoting 
the recognition of prior learning." 

There are two basic areas of possible development in QA of lifelong learning: how 
external QA procedures deal with the identification of lifelong learners' profiles and 
expectations? And how external QA can be in line with HE systems which conceive new 
educational programmes responding to new lifelong learning schemes? 
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For QA agencies, lifelong learning goes beyond traditional learning. Lifelong learning 
relates to agendas on diversity and inclusivity and might involve work-based learning, 
formal access to higher education and continuing professional development. 

As regards the second question, external QA procedures should be developed jointly 
with the Qualifications Frameworks for lifelong learning. This is an important issue that 
need to be clarified if agencies have to develop procedures in this field. Indeed, the 
procedures have to take into account the existence of a set of non-degree programmes, 
non-formal or informal learning or the combined educational programmes in which the 
work-based learning is relevant. 

c) QA and online education 

Half of the respondent agencies cover this area, although they do not consider this type 
of external QA as a priority. In fact, they do not mention this type of external QA for 
higher education as one of the most effective and successful practice. 

Although two QA agencies (ECBE and OAR) report relevant practices connected with 
this ENQA priority - QA for online programmes and online education, the survey does 
not provide sufficient information on how different agencies include the quality of online 
education in their QA schemes for programme or institutional review. Only one answer 
states that the agency's procedure is designed to generate a positive impact for students 
in their teaching and learning. 
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CHAPTER 3 


EXAMPLES OE GOOD PRACTICE 


The survey invited agencies to identify a current good practice (only one per agency) 
in external quality assurance that could impact positively on the development and 
implementation of their activities in the future. In general, the answers to that question 
do not offer examples on long term and consolidated procedures, but on relatively new 
strategies and tasks for which QA agencies are supposedly most satisfied with. To 
some extent, this gives an opportunity to see on what type of activities the agencies 
concentrate part of their energy and interests. 

Making a clear classification of the good practices reported in the survey was not 
easy, especially because some of them cover different objectives and areas. However, the 
answers suggest the following distribution: 

a. practices regarding external QA procedures; 

b. practices which enhance stakeholders' involvement in QA of HE; 

c. practices aiming at improving the infrastructure and resources of agencies; and 

d. other developments. 

It is noteworthy that the enhancement of internal Infrastructure could be considered as a 
good practice in internal quality assurance and development of agencies, rather than in 
external quality assurance. 

The following chart shows the areas of good practice according to the above described 
distribution. 

Table 8: Distribution of good practices 


3.1 EXTERNAL EVALUATION AND ACCREDITATION PROCEDURES 

Eleven agencies reported external QA procedures as a good practice. The variety of 
mechanisms is important. The nomenclature for the external QA procedures, used 
both at programme and institutional level, includes accreditation, certification, audit, 
validation and recognition^. 

3.1.1 AT PROGRAMME LEVEL 

Three agencies reported good practices in professional-oriented study programmes, thus 
demonstrating a willingness to improve the connection between higher education and 
the professional demands for graduates through particular external QA procedures. 

Two of them accredit groups of study programmes included in a certain discipline. 
According to them, the benefits of such practice are the achievement of a higher focus 

8 The report does not give a definition of these procedures on the grounds that this was not requested in the survey and 
that the interpretation of the terms differs from one country to another. 



Areas of good practice 


External QA procedures 48% 
Stakeholders involvement 26% 


Agency infrastucture 17% 
other developments 9% 
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on quality issues and the establishment of connections between different units and 
programmes. They also believe that this approach improves the promotion of quality 
culture within higher education systems through the publication of evaluation reports 
which cover the assessment results in comparable programmes at the same time. 

The examples of accreditation (validation) of collaborative programmes in Ireland or 
the accreditation of adult education programmes as part-time modularised studies in 
Denmark show how agencies are extending their accreditation procedures to new types 
of higher education. 

Finally, as previously stated in the report, certification of good practice in 
internationalisation at programme level and recognition of good practice in HE are 
carried out by two agencies (NCPA and NVAO). 

3.1.2 AT INSTITUTIONAL LEVEL 

The survey collects a good practice in institutional Audit which, as a general purpose, 
supports the internal QA system. 

Another example characterised by a certain "back to basics" strategy was also 
detected. This is the case of institutional evaluations recovering the original values, for 
example by highlighting the external QA momentum as a promoter of dialogues between 
external reviewers and university responsible managers. 

In the same direction, another example is provided by an agency which promotes 
meetings with higher education institutions in order to impulse the follow-up actions and 
good practice dissemination. 

3.2 INVOLVEMENT OF STAKEHOLDERS 

Practices attempting to increase the participation of different stakeholders in external 
QA procedures could be observed throughout the survey, but at this stage, the agencies 
had the opportunity to highlight one good practice. One quarter considered their work 
with stakeholders as sufficiently relevant to be reported as a good practice. 

The collaboration with students, as stakeholders, has been recorded in the survey 
and provides two innovative cases: a) the promotion of students' involvement as a 
global strategy for external QA and quality enhancement in higher education and b) the 
creation of an Advisory Committee in an agency composed by students. 

The work with experts in education and the creation of focus groups composed by 
representatives of academic communities has been stated in the survey. 

In general, respondents mainly gave examples of good practices connecting 
stakeholders with the agency's governance structures. 

In Croatia, an interesting case is the involvement of the civil society working in the field 
of higher education, through a NGO representative in the Accreditation Council of ASHE. 
Without voting rights, this NGO representative is seen by the agency as a possibility to 
increase public monitoring of decision-making by the Accreditation Council. There is 
another example provided in the survey by HQAA where trade unions are involved in the 
Board of the agency. 

The inclusion of employers in the process of control over the quality of HE was 
mentioned by one agency and could be seen as a similar process of connecting social 
interests with the work of the agencies. 

Involving international stakeholders in the governance structure is referred to as 
another good practice whereby international members (who can be considered as 
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international stakeholders) are included in the accreditation body. This strategy is done 
with the purpose of increasing the visibility of the agency's independence. 

Finally, it is worth highlighting the interest of AQA to collect feedback on a regular 
basis from HEIs, peer experts and other stakeholders. This feedback feeds into the 
evaluation of AQA and facilitates the development of its QA activities. 

3.3 INTERNAL INFRASTRUCTURE OF AGENCIES 

In connection with the growing importance of the use of information, there is an effort to 
generate electronic platforms and software tools to deal with the external QA procedures 
of agencies. The use and management of information is seen as an important element for 
an efficient organisation of external evaluations and for public information purposes. 

There is also a good practice of developing pools of experts (including students) for 
accreditation and external quality control. It was not reported in the survey, but this 
practice gains attention for different reasons. The international cooperation in this matter 
seems still underdeveloped. 

3.4 OTHER ELEMENTS 

Two cases were grouped in this category. In the first case, an agency provides advice 
and support to HE institutions for the design and implementation of their internal QA 
systems. The good practice emphasises the importance of the well-known Quality 
Assurance scheme PDCA (Plan, Do, Check, Act); in other words, the agency cooperates 
with the HE institutions to design their internal QA system. 

Developing consultation services and conducting accreditation procedures at the 
same time (i.e. giving advice and then evaluating what has been advised) need to be 
treated carefully, since in many countries there is a tendency to avoid mixing the two 
functions to avoid conflict of interest. 

The second case is the development of a set of quality indicators for the whole HE 
system. 

In conclusion, the reported good practices draw up a group of strategies focused 
on the promotion of the quality culture within HEIs, and characterised by a greater 
participation of stakeholders and the civil society in general. The exchange and access to 
reliable information is at the heart of that group of strategies. 
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CHAPTER 4: 

PROSPECTIVE PROCEDURES AND 
EUTURE AREAS OE IMPROVEMENT 


4.1 INTRODUCTION OF MAJOR CHANGES ON OA PROCEDURES 

Sixty percent of the respondents (15 agencies) answered yes to the question "Does your 
Agency plan to introduce major changes in external QA procedures in the future?" and 16 
provided a description of the prospective change or procedure. 

Besides the amalgamation of QA agencies in Austria and Ireland driven by political 
and regulatory changes, the main focus on prospective procedures is to reinforce and/or 
reformulate the external QA procedures at the institutional level. 

It is important to make clear that the use or the introduction of institutional 
QA procedures does not necessarily mean that external QA procedures for study 
programmes are inexistent. Actually many countries are doing both types of evaluations 
or have integrated both dimensions in comprehensive schemes. 

The following chart shows that programme and institutional QA are not two different 
worlds but integrated realities. There is a tendency to emphasise the procedures at 
institutional level, but this seems to be connected with the purpose of looking for the 
most effective and efficient QA procedures that would promote the enhancement at both 
levels, in particular at programme level. 


Table 9 



Introduction of major changes in external 
QA in the future 

I Institutional QA 34% 

B Amalgamation of QA agencies 33% 

B Integrated institutional - programme QA 13% 

B others 13% 

Programme QA 7% 


In fact, if we look at the current relevant procedures reported in Part 1 of the survey, 
it is possible to observe the tendency to report relevant external QA procedures both 
at programme and institutional level (14 agencies), while 11 agencies reported having 
relevant procedures in one of those dimensions (seven for programmes and four for 
HEIs). 

In the first part of the survey, the agencies were invited to inform about three relevant 
QA procedures. Given that many respondents mentioned only one or two best practices, 
it can be presumed that the first procedure was considered as the most relevant one, 
with a decreasing relevance for the second and third procedures. The following chart 
shows a certain prevalence of external QA procedures for study programmes. 
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14 

12 



Procedure 1 Procedure 2 Procedures 


Table 10 (in number of procedure) 

Relevant procedures 
on external QA 

B Programme level 
B Institutional level 
B Other developmentsr 


4.2 PROPOSED AREAS FOR IMPROVEMENT AND DEVELOPMENT 

The survey proposed nine different pre-defined areas for possible improvement or 
development (cf. chart below). The number of areas selected by agencies does not follow 
a clear pattern: 10 of them chose two or three areas only, and 12 reported four or more 
areas. The final outcome shows a great homogeneity in the distribution of interests. 
While the area "QA of HE and research" scores low, the rest of the areas score quite 
similar. 
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Table 11 (in number of agencies) 

Areas where there 
is a major need for 
development 


Even if certain priorities are identified among the answers from the 12 agencies who 
selected two or three areas, it is not possible to identify a clear pattern. 

However, for the seven agencies which selected only two areas, the highest interest is 
on the "use of data in QA". 

The answers given by the group of four agencies selecting three areas were slightly 
different. Those agencies are more interested in the "development of international 
benchmarks for QA" and in the "international recognition of national QA processes". In 
this second group, only one agency declared interest in the "use of data in QA". 

In conclusion, the answers show that the five areas where agencies see a major need 
for development in the coming years are: use of data in QA, international recognition of 
national QA, international benchmarks for QA, university primary responsibility on QA 
and use of indicators. 
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ANNEX: THE QUALITY 
PROCEDURES QUESTIONNAIRE 



The objective of this survey is to promote dialogue with different representatives on the relevance and impact of different external QA 
procedures and their future evolution. Therefore, it is of utmost importance that the questionnaire is completed in consultation with as 
many members of your Agency (staff, Board/Council members, etc.) and stakeholders as possible. 

This project is funded by the European Commission through its Lifelong Learning Programme. 

The proposed survey begins with an introduction relating to general information about the agency and its relationship with ENQA. It also 
inquires about the methodology used for working with the questionnaire. 

The main body of the questionnaire is divided into two parts. The first part is devoted to obtain a general picture of what is happening about 
external QA procedures. To that end, the agencies are inquired about their current and future activities in relation to the ENQA priorities and 
other complementary elements. 

The second part of the questionnaire tries to identify, in depth, individual features of good practice (implemented or expected to be applied in 
the near future). This part of the survey will help ENQA in mapping developed/advanced procedures. 

Qne of the results of this project is the creation of an online tool for agencies' quality procedures. This database on the ENQA website will 
gather good and efficient quality procedures to be shared among QA professionals in Europe and beyond. The responses will be most valuable 
as they will contribute to the creation of this web resource. 

The survey is being conducted by an ENQA project group composed of: 

- Josep Grifoll, Chair, AQU and ENQA Board member 

- Achim Hopbach, GAC and ENQA President 

- Helka Kekalainen, FINHEEC and ENQA Vice-President 

- Christina Rozsnyai, CEE Network 

- Todor Shopov, CEE Network 

- Nathalie Costes, ENQA, Secretary 

The responses will be analysed by the Project Group, and presented in the Third Report on Quality Procedures in European Higher Education, 
to be published by November 201 1 . The final report of the project will be considered for the preparation of an ENQA position paper to be 
submitted to the ministerial meeting of April 2012 in Bucharest, and will also be exploited by a wider audience interested in enhancing a 
quality culture in the field of higher education. 

Your responses will not be used in any ENQA coordinated review of your agency or for any purpose other than that stated above. 

The closing date for this survey is 1 5 April 201 1 . 

If you have any questions about the survey please contact Nathalie Costes (nathalie.costes@enqa.eu). 



1 . Name of your Agency 
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2. What is your Agency's relation to ENQA 

o Full member 

Candidate member 

o Associate 

Affiliate 

3. Name of respondent and contact information 

Name: | | 

E-mail: | | 

Address: 

City/Town: 

State/Province: 

ZIP/Postal Code: | | 

Country: 

4. Grant/refusal of permission to ENQA to publish your responses: 

o I give permission for my Agency to be identified with the responses I give in the final report and web resource. 

I do not give permission for my Agency to be identified with the responses I give in the final report and web resource (your responses 
will be reported anonymously). 


Methodology 


1. How did you answer the questionnaire? 

a) Did you involve the QA | 
agency staff? (Please give 
details) 

b) Did you involve the | 

Agency Board/Council? 

(Please give details) 

c) Did you involve other | 
agents (e.g. stakeholders)? 

(Please give details) 


Part 1 - QA procedures in relation to ENQA priorities and other complementa... 


A: ENQA priorities 

The position of your Agency’s activities regarding the priorities of ENQA, listed in the bullet points below. 
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1. Which area does your Agency cover (i.e. in which area does it conduct external QA 
evaluation)? 

□ QA and enhancement mechanisms 

□ QA of transnational programmes 

□ Lifelong learning and QA (Please specify > question 2) 

I I QA in online education 

□ Qualifications frameworks & QA 

□ QA and learning outcomes 

□ Excellence in higher education (Please specify > question 3) 

□ QA and Employability 

I I QA and public information on HE 

2. If you answered "Lifelond learning and QA", please give us a brief definition of what 
you mean by lifelong learning in your context? (e.g. HE programmes for leaners over 25 
years old, non-degree HE programmes for adults, etc.) 



3. If you answered "Excellence in higher education", please define the meaning of 
excellence in HE in your context and what are the purposes of connecting QA and 
excellence in HE? 
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4. Which area do you consider as a priority in your Agency? Rate each area 1 to 5. 



1 Not at all a priority 

2 

QA and enhancement 
mechanisms 
QA of transnational 
programmes 

o 

o 

o 

o 

Lifelong learning and QA 

o 

o 

QA in online education 

o 

o 

Qualifications frameworks & 
QA 

o 

o 

QA and learning outcomes 

o 

o 

Excellence in higher 
education 

o 

o 

QA and employability 

o 

o 

QA and public information 
on HE 

o 

o 


3 

4 

5 A very important 
priority 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 


5. Considering the priorities of ENQA, in which area are you currently most effective and 
successful? 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 


QA and enhancement mechanisms 
QA of transnational programmes 
Lifelong learning and QA 
QA in online education 
Qualifications frameworks & QA 
QA and learning outcomes 
Excellence in higher education 
QA and Employability 
QA and public information on HE 


Part 1 - QA procedures in relation to ENQA priorities and other complementa... 


A: ENQA priorities 

Identification of current QA procedures 
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1. Please identify a maximum of 3 relevant QA procedures implemented by your Agency 
and connected with the priorities of ENQA? 


2. Procedure 1 : 
A) 


Name of the implemented 
procedure: 

Please give a brief 
description of the 
procedure: 

If possible, please give an 
example of the public 
report for that procedure 
(please provide a link): 


3. Procedure 1 : 

B) Name(s) and contact details of the responsible project manager(s) in that/those area 


Procedure 1 


(s)? 
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4. Procedure 1 : 

C) Institutions invoived 

□ A) all in the system 

□ B) some of them (how many, in % of total?) 

□ C1) types of institutions: private universities 

□ C2) types of institutions: public universities 

□ C3) types of institutions: private HEIs such as university colleges, applied universities etc. (please specify) 

□ C4) types of institutions: public HEIs such as university colleges, applied universities etc. (please specify) 
If you chose B, C3 and/or C4, please specify: 



5. Procedure 1 : 

D) Is the procedure implemented in cooperation with another QA agency? 

O 

Quo 

6. Procedure 1 : 

E) Do external stakeholders participate in the implementation of the external QA 
procedure? 



Yes 

No 

Students 

o 

o 

Employers 

o 

o 

Public authorities 

o 

o 

Trade unions 

o 

o 


Others (please specify) 

I 1] 
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7. Procedure 1 : 

F) Expected benefits for stakeholders (rate 1-5): This table should help the QA agency to 
reflect about the results of the procedure in providing benefits for the stakeholders. One 
or more benefits in axis x can be considered for the stakeholders in axis y. 


Students 
Employers 
Public authorities 
Trade unions 
Others (please specify) 


Development 

Better Increased of new Better 

Development 

connection relation indicators Promotion connection 

of the Improvement Improvement 

between between about the of between 

primary of teaching of public 

social teaching- quality of HE innovation national and 


Other 

benefits 


Donsibility 

HEI’son 

QA 

interests 
and HE 
outcomes 

and learning 
processes 

learning 

and 

research 

Enhancement 
of HE 

organisations 
and structures 

in higher 
education 

information 
on HE 

international 
standards for 
HE 

(please 

specify) 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 


Others / Other benefits? (please specify) 



8. Procedure 1 : 

G) Do you have any particular action for promoting the benefits for international 
stakeholders? 

0 

Qno 

If yes, please describe: 

1 
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9. Procedure 1 : 

H) Expected benefits for HE institutions (rate 1-5) 


Development of the 
primary responsibility of 
HEI’s on QA 

Better connection between 
social interests and HE 
outcomes 

Improvement of teaching 
and learning processes 
Increased relation between 
teaching-learning and 
research 

Development of new 
indicators about the quality 
of HE 

Enhancement of HE 
organisations and structures 
Promotion of innovation in 
higher education 
Improvement of public 
information on higher 
education 

Better connection between 
national and international 
standards for HE 


1 Poor 

o 

2 

o 

3 

o 

4 

o 

5 Excellent 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 


Other benefits (please specify): 



Procedure 2 

10. Procedure 2: 
A) 

Name of the implemented 
procedure: 

Please give a brief 
description of the 
procedure: 

If possible, please give an 
example of the public 
report for that procedure 
(please provide a link): 
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11. Procedure 2: 

B) Name(s) and contact details of the responsible project manager(s) in that/those area 
(s)? 

i] 

12. Procedure 2: 

C) Institutions involved 

□ A) all in the system 

□ B) some of them (how many, in % of total?) 

□ C1) types of institutions: private universities 

□ C2) types of institutions: public universities 

□ C3) types of institutions: private HEIs such as university colleges, applied universities etc. (please specify) 

□ C4) types of institutions: public HEIs such as university colleges, applied universities etc. (please specify) 


If you chose B, C3 and/or C4, please specify: 



13. Procedure 2: 

D) Is the procedure implemented in cooperation with another QA agency? 

O 

Quo 
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14. Procedure 2: 

E) Do external stakeholders participate in the impiementation of the externai QA 
procedure? 


Students 
Employers 
Public authorities 
Trade unions 
Other (please specify) 


Yes 

o 

No 

o 

o 

o 

o 

o 

o 

o 


3 


15. Procedure 2: 

F) Expected benefits for stakehoiders (rate 1-5): This tabie shouid help the QA agency to 
reflect about the results of the procedure in providing benefits for the stakehoiders. One 
or more benefits in axis x can be considered for the stakehoiders in axis y. 

Development 

Better Increased of new Better 

relation indicators Promotion connection 


Development 


of the 
primary 
responsibility 
of HEI’s on 
QA 


connection . 

Improvement Improvement 

between between about the of between 

of public 

quality of HE innovation national and 

information 


social 
interests 
and HE 
outcomes 


of teaching 
and learning 
processes 


teaching- 

learning 

and 

research 


Enhancement in higher 
of HE education 
organisations 
and structures 


on HE 


international 
standards for 
HE 


Other 

benefits 

(please 

specify) 


Students 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Employers 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Public authorities 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Trade unions 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Others (please specify) 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 


Others / Other benefits? (please specify) 
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16. Procedure 2: 

G) Do you have any particular action for promoting the benefits for international 
stakeholders? 

o 

0 

If yes, please describe: 

1 


17. Procedure 2: 

H) Expected benefits for HE institutions (rate 1-5) 


Development of the 
primary responsibility of 
HEI’s on QA 

Better connection between 
social interests and HE 
outcomes 

Improvement of teaching 
and learning processes 
Increased relation between 
teaching-learning and 
research 

Development of new 
indicators about the quality 
of HE 

Enhancement of HE 
organisations and structures 
Promotion of innovation in 
higher education 
Improvement of public 
information on higher 
education 

Better connection between 
national and international 
standards for HE 


1 Poor 

o 

2 

o 

3 

o 

4 

o 

5 Excellent 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 



Procedure 3 
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18. Procedure 3: 

A) 

Name of the implemented | | 

procedure: 

Please give a brief | | 

description of the 

procedure: 

If possible, please give an | | 

example of the public 
report for that procedure 
(please provide a link): 

19. Procedure 3: 

B) Name(s) and contact details of the responsible project manager(s) in that/those area 
(s)? 

20. Procedure 3: 

C) Institutions involved 

□ A) all in the system 

□ B) some of them (how many, in % of total?) 

□ C1) types of institutions: private universities 

□ C2) types of institutions: public universities 

□ C3) types of institutions: private HEIs such as university colleges, applied universities etc. (please specify) 

□ C4) types of institutions: public HEIs such as university colleges, applied universities etc. (please specify) 


If you chose B, C3 and/or C4, please specify: 



21. Procedure 3: 

D) Is the procedure implemented in cooperation with another QA agency? 

O 

Quo 
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22. Procedure 3: 

E) Do external stakeholders participate in the impiementation of the externai QA 
procedure? 


Students 
Employers 
Public authorities 
Trade unions 
Others (please specify) 


Yes 

o 

No 

o 

o 

o 

o 

o 

o 

o 


3 


23. Procedure 3: 

F) Expected benefits for stakehoiders (rate 1-5): This tabie shouid help the QA agency to 
reflect about the results of the procedure in providing benefits for the stakehoiders. One 
or more benefits in axis x can be considered for the stakehoiders in axis y. 

Development 

Better Increased of new Better 

relation indicators Promotion connection 


Development 


of the 
primary 
responsibility 
of HEI’s on 
QA 


connection . 

Improvement Improvement 

between between about the of between 

of public 

quality of HE innovation national and 

information 


social 
interests 
and HE 
outcomes 


of teaching 
and learning 
processes 


teaching- 

learning 

and 

research 


Enhancement in higher 
of HE education 
organisations 
and structures 


on HE 


international 
standards for 
HE 


Other 

benefits 

(please 

specify) 


Students 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Employers 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Public authorities 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Trade unions 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Others (please specify) 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 


Others / Other benefits? (please specify) 
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QUALITY PROCEDURES PROJECT “VISIONS FOR THE FUTURE" 


24. Procedure 3: 

G) Do you have any particular action for promoting the benefits for international 
stakeholders? 

o 

O 

If yes, please describe: 


3 


25. Procedure 3: 

H) Expected benefits for HE institutions (rate 1-5) 


Development of the 
primary responsibility of 
HEI’s on QA 

Better connection between 
social interests and HE 
outcomes 

Improvement of teaching 
and learning processes 
Increased relation between 
teaching-learning and 
research 

Development of new 
indicators about the quality 
of HE 

Enhancement of HE 
organisations and structures 
Promotion of innovation in 
higher education 
Improvement of public 
information on higher 
education 

Better connection between 
national and international 
standards for HE 


I Poor 

o 

2 

o 

3 

o 

4 

o 

5 Excellent 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 



Part 1 - QA procedures in relation to ENQA priorities and other complementa. 


B: Policy Evaluation 
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1. What is the position of the ministry/national authorities on the relevance of external 
QA processes for promoting and safeguarding academic mobility? 

n 

2. Has your Agency received instructions in order to develop procedures for promoting 
academic mobility? 

O 

Quo 

If yes, what kind of development has been put in place? 

ij 

3. What is the position of the ministry/national authorities on the appropriateness of QA 
agencies to deal with the objective of equitable access to Higher Education? 

il 

4. Has your Agency received instructions in order to develop procedures for 
evaluating/promoting equitable access? 

o 

0 

If yes, what kind of development has been put in place? 

1 1] 


Part 1 - QA procedures in relation to ENQA priorities and other complementa... 


C: Prospective procedures and future areas of improvement 

1. Does your Agency plan to introduce major changes in external QA procedures in the 
future? 

O 

Qno 


Page 15 




45 


QUALITY PROCEDURES PROJECT “VISIONS FOR THE FUTURE 


2. If yes, please give details on what kind of procedures are expected to be introduced: 

Name of the prospective | ~[ 

procedure 

Main purpose | | 

Contact person | | 

3. In which areas does your Agency think there is a major need for improvement and 
development? 

□ Use of data in QA (collection, management...) 

□ Use of indicators in your QA processes 

□ Stakeholders’ involvement in QA processes 

□ QA and new paradigms for teaching and learning (e.g. student as co-producer) 

□ QA of HE and research 

□ Cost-effectiveness of external QA 

□ Promotion of university primary responsibility on QA 

□ Development of international benchmarks for QA development 

□ International recognition of national QA processes 


Part 2 - Identification of a current good practice (one per agency) 


This part is specifically addressed to describe, in depth, a good practice in external QA that could impact positively on 
how QA agencies develop and implement their activities in the future. 

Good practice is defined as a positive action that must: 

- be successful: it provides positive results for the objective of enhancing a quality culture in the field of higher education; 

- be innovative: something different has been implemented. Innovative means providing new or different solutions to 
existing ones in the region or sector. Solutions can be completely new or incorporated by transference from other 
contexts. Innovation can be found in the process (measures, contents, methods, approaches, tools), in the object (new 
areas of interest, new social groups) or in the context (adaptation or improvement on the current conditions, starting-up of 
networks); 

- have a possible multiplying effect or transference to other areas: it should be visible, communicable and 
shareable (dissemination) and/or integrated and applicable to systems and regulations; 

- be sustainable: it is self-supporting both currently and in the future. 

1 . Name of the practice /procedure / prospective procedure: 

I ^ 
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2. Name of the responsible manager of the practice (if already identified) / position at the 
QA agency: 

3. Description of the procedure: 

(If possible, please provide the web link where public information about the procedure is 
available) 

4. What positive results do you expect from that procedure with respect to the 
enhancement of a quality culture in your HE system? 



5. To what extent is this procedure providing new solutions to existing QA procedures 
in the region or in the sector? 

Note: innovation can be found in the processes (measures, contents, methods, 
approaches, tools), in the object (new areas of interest, new social groups) or in the 
context (adaptation or improvement on the current conditions, starting-up new 
networks) 
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6. To what extent is the procedure transferable to other areas or realities? 



7. Is the practice sustainable currently and in the future? 

o 

O 

Please explain why: 
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